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ABSTRACT 

The Client Observation Checklist (COC) was developed 
to evaluate Project ADAPT' s intervention m three behavioral areas: 
bathing; dressing; and socialization. Project ADAPT is de^igped to 
provide services to meet the needs of chronically mentally ill 
residents of nursing homes. Specifically, the project provides staff 
trained to work with the mentally handicapped and therapeutic 
intervention to raise motivation levels. Fifteen clients of Project 
ADAPT at each of two facilities were selected for observation. 
Observations were made during the last 2 weeks of March 1984, before 
Project ADAPT had begun at either location. Two pairs of independent 
raters averaged 94.44% and 95.37% agreement, respectively, 
establishing very Figh reliability for the COC. The validity of the 
COC was determined by correlating subscale totals with case manager 
ratings on the Treatment Setting Determination Survey. Despite two 
sources of potential error variance, mtercorrelations between 
baling and dressing subscales/items all exceeded 0.70, establishing 
the validity of the COC. The COC required an average of 19.67 minutes 
to administer. The non-signif icant tendency for clients at one 
facility to have a higher level of functioning is described as 
consistent with the nonequivalent control group design used for 
evaluation. Two data tables and the COC are included. (TJH) 
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Summary 

The Client Observation Checklist (COC) was developed to evaluate Project 
ADAPT's intervention in three behavioral areas; bathing, dtessing > an:l 
socialization. Two pairs of independent raters averaged 94.44% and ^5.3"% 
agreement, establishing very high reliability for the COC. Validity of the COC 
was determined by correlating subscale totals with Caseiranager r^ ? ng^ on the 
Treatment Setting Determination Survey. Despite two sources of potent: 4 a) ^ncr 
variance, intercorrelat ions between bathing and dressing subscale - 'ite«v?s aU 
exceeded .70, establishing the validity of the COC. Th* COC re<?ulr<?^ an 
average of 19.67 minutes to administer; but, observers had to vait 17.87 
minutes before rating the typical client. Facilities differed concerning the 
proportion of time that raters spent waiting to see clients. The 
nonsignificant tendency for clients at one facility to have a ntgher level of 
functioning is described as consistent with the Nonequi valent Control Group 
Design employed for the evaluation. 
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Development of the Client Observation Checklist: Reliability, Validity, 
and Differences between Facilities Chosen to Receive Project ADAPT 



In March, 1984, the St. Louis Regional Community Placement Program (CPP) 
selected four facilities to receive services from the Project ADAPT Institute. 
Project ADAPT is designed to provide two services to meet the specialized needs 
of chronically entally ill residents of nursing homes: (1) staff whu are 
trained to work with the mentally handicapped; and, (2) therapeutic 
intervention to raise .notivation levels. 

Project ADAPT had already been delivering services to residents at two other 
facilities since 1983. Client behaviors which Project ADAPT staff reported 
they had most frequently addressed at one of these facilities served as the 
basis for the proposed evaluation (Fitz, 1984A). The Client Observation 
Checklist (COC), which is the focus of the current import, was designed to 
record three client behaviors (bathing, dressing, and socialization) which 
could not be obtained by archival data. The final COC form which was used is 
given in Appendix A. 

When Project ADAPT staff work with a client's activities of daily living (i.e., 
walking, eating, bathing, dressing, hygiene, continence, socialibility) , they 
select a specific activity, divide it into component parts, and address one 
component at a time. The COC is intended to be a raeasur ment which parallels 
the perspective of this approach. 

The COC components for each subscale are modifications of Project ADAPT's own 
evaluation tools. Originally, the author had intended to use four items 
paraphrased from the Project ADAPT hygiene checklist ("washed independently"; 
"showered independently"; "combed own hair"; and, "shaved independently or put 
on own make-up"). However, CPP staff raters discovered that few, if any, 
clients chosen for observation showered and felt that the last two items would 
introduce strong sex differences. Therefore, CPP staff developed the COC 
"Bathing Checklist", which requires ratings of whether a client bathes each of 
four body areas. The Project ADAPT dressing checklist was modified in the 
following ways to form the COC "Dressing Checklist": the "cooperati veness" and 
"dresses appropriately" items were omitted since they seemed too subjective; a 
"selected own clothes" item was added; and, "snapped and hooked" were included 
with "zipping and buttoning". Since the Project ADAPT socialization items 
seemed to be nonspecific, CPP staff decided to use a structured interview while 
observing bathing and dressing. Responses to this interview provide scores for 
the COC "Socialization Checklist". 

Approximately 159 clients at 4 facilities were scheduled to receive Project 
ADAPT services. Fifteen clients at each of two facilities were selected for 
observation. They were chosen randomly, with the restriction that either the 
CPP or facility nurse had rated dressing behavior as "independent" for 2 
clients, as "needing supervision" for 7 clients, and as "needing to be dressed" 
for 6 clients at each facility. Project ADAPT services began on April 1, 1984 
at Facility 1 and will begin later at Facility 2. The current report analyzes 
data from observations made during the last two w^eks of March, 1984, before 
Project ADAPT had begun at either location. 
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Reliability 

According to Crano and Brewer (1973, p. 153), "The basic question of 
reliability may be most simply stated as follows: Do the ratings of two or 
more observers who have witnessed the same event(s) coincide to a major 
degree?" A measure is reliable to the extent that it is consistent. Since it 
would be much too time consuming and expensive to use a large number of 
multiple observations for a correlational analysis, this investigation had the 
goal of establishing the reliability of the COC from extremely high agreement 
in a few observations. The following steps were taken to reach high agreement: 

1. The previously-described changes in the COC were made after 
discussions between four CPP staff who had used it for trial 
observations: the author, a Psychiatric Nurse (R.N.) assigned 

to Program Development and Evaluation (PDE); a Psychiatric Social 
Worker (M.S.W.) assigned to PDE; and the CPP Program Director 
(M.S.W.); 

2. After the final form of the COC was agreed upon, three raters 
were selected: the Psychiatric Nurse, the Psychiatric Social 
Worker, and a Research Assistant (B.A. in Communications); 

3. The nurse became the primary rater, which meant the other two 
would rate by themselves only after rating a client with her and 
disagreeing 0 or 1 point on the COC; and, 

4. After rating six clients independently, the social worker and 
research assistant would again rate with the primary rater until 
sufficient agreement was reached. 

For each rating by two people, a percentage agreement (PA) score was computed 
as follows: 

PA = (A/T) X 100 

where A is the sum of agreement scores on all items, and T^ is the total 
possible agreement. Since each item is rated 0, 1, or 2, the agreement score 
for each item ranges from 0 to 2 per item. With 18 items, the value of T^ is 
36. 

The Psychiatric Social Worker observed two clients with the primary rater. 
They had 4 points of disagreement on the first observation and 0 points of 
disagreement on the second, for a mean PA score of 94.44%. The Research 
Assistant's three observations with tne primary rater had 3, 2 and 0 points of 
disagreement, for a mean PA = 95.37%. With both PA mean scores over 90%, the 
Client Observation Checklist is clearly a highly reliable instrument. 
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Validity 

Oskamp (1977, p , 38) defines validity as the "... accuracy or correctness of 
measurement. 11 To the extent that ratings on a scale correspond to scores on an 
established scale, the measurement is valid. Validation for the COC was 
determined by correlating its subscales with the ratings on the Treatment 
Setting Determination Survey (TSDS, Keck, Ahr, & Harris, 1981), which was 
completed by client Casemanagers the same month as the COC rating. The TSDS 
was designed to determine if mental health clients of the Missouri Department 
of Mental Health are placed in a facility which matches their needs for care. 
Since CPP Casemanagers complete it on a quarterly basis, they are quite 
familiar with the instrument. While the TSDS rating options are in such broad 
categories that the instrument would not be expected to pick up subtle changes 
resulting from Project ADAPT interventions, it does provide an extremely useful 
measurement against which to test the validity of the COC. 

Correlations were computed between COC summary scores for Bathing, Dressing, 
and Socialization items (referred to as BASUM, DRSUM, and SOSUM. respectively), 
TSDS summary scores for Activities of Daily Living (ADL), behavior problems, 
and psychiatric problems, and TSDS individual items for ADL, health, Global 
Assessment Scale (GAS) and whether the client has a friend or relative likely 
to visit (VISIT). Table 1 pro/ides the correlation matrix between BASUM, 
DRSUM, SOSUM and the six ADL items from the TSDS. Several important patterns 
emerged from the correlational analyses: 

1. Baching and dressing form a very cohesive cluster as indicated by: 

a. the high correlation between BASUM a - DRSUM (£ - .81) on the COC; 

b. the high correlation between TSDSBATH and TSDSDRES ( r_ - .91); and, 

c. the high correlations (across scales) between BASUM and DRSUM and 
the TSDS bathing and dressing items (.70 < _r's < .80); 

2. Socialization is related to several other variables as indicated by: 

a. the moderately high correlation between SOSUM and BASUM (r - .47) 
and DRSUM (_r - .43); 

b. the strong correlation between SOSUM and TSDSEAT (_r - .60); and, 

c. the moderately high correlation between SOSUM and GAS (not shown in 
Table 1 , r^ * .47). However, GAS had a stronger relationship with 
BASUM Or - .68) and DRSUM (jr = .55) and SOSUM had a surprising 
absence of a relationship with VISIT (j: - .00); 

3. The high correlation between fecal and urinary incontinence (_r = .90) 
suggests incontinence is a unitary dimension. 

The correlation matrix in Table 2 was also done separately for the 15 clients 
at each facility, which allowed a comparison of the validity of the TSDS for 
each Casemanager. For one Casemanager, the correlations between the COC and 
TSDS bathing and dressing items were quite high (.86 < r/ s < .87). However, 



ERLC 



r 

u 



4 



for the other Caseraanager, correlations for TSDSBATH could not be computed due 
to absence of variability and correlations between TSDSDRES and BASIM Cr = .48) 
and DRSUM (r ■ .27) were much lower. This suggests that the TSDS scale may 
have more or less reliability, depending on the person completing it. 

In conclusion, the COC has been shown to be a valid instrument, especial]/ for 
the bathing and dressing subscales. The correlations between BASUM, DRSUM, 
TSDSBATH, and TSDSDRES include potential error variance from two sources: 
differences between scales; and, differences between observers. The fact that 
the correlations are nevertheless quite high indicates that strong confidence 
can be given to these dimensions. 



Differences between Two Facilities Chosen to Receive Project ADAPT 

Time for observation . On the average, raters spent 37.60 minutes for each 
observation — 17.87 minutes waiting to see the client end 19.67 minutes 
actually observing the client bathe and dress while conducting the interview. 
To the extent that a facility is efficient and well-organized, we would expect 
that raters would spend less time on the COC. There was no difference in the 
total time spent on the POC at Facility 1 (M - 33.73 minutes) and Facility 2 
(M - 41.47 minutes), _t(28) < l.C. However, there was a large difference in the 
way raters 1 time was apportioned: at Facility 1, raters averaged 27.48% of 
their time waiting to see the client, and, at Facility 2, they spent 47.73% of 
their time waiting, t_(28) - 2.54, £ < .02. Aides at Facility 2 frequently 
commented to raters that clients could not begin their baths because there was 
so little hot water. 

CQC subscales . For each of the three COC subscal^s, there was a nonsignificant 
tendency for residents at Facility 2 to be slightly higher functioning. The 
t-test for these subrcale items and the chi-square tests for individual item 
contrasts between facilities are given in T able 2. Since only 3 of 18 
chi-square comparisons teach conventional levels of significance, we can 
conclude that the differences between the facilities are minimal. This is 
consistent with the Scheffe tests on TSDS ratings which found Facilities 1 and 
2 did not differ on overall ADL scores or dre»^in^ scores, but d\d have a small 
but significant difference on bathing scores (Fitz, 1984B). This TSDS 'ita, 
however, was for all clients at the two facilities rather than for the ubsot 
of 15 chosen for COC comparisons. 



Conclusions 

The CCC was developed to use for a contrast of client functioning at two 
facilities, one of which received Project ADAPT services before the othe^. 
Clients were pretested at both facilities and will be tested again at two 
months and four months. Therefore, the research plan corresponds to Carapbel 1 
and Stanley's (1963, p. 47) Design 10, The Nonequlvalent Control Group Design , 
which " • • • involves an experimental group and a control group both given i 
pretest and a posttest, but in which the control group and the experimental 
group do not have pre-experimental sampling equivalence." 
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As they point out, 

The more similar the experimental and the control groups are in their 
recruitmenc, and the more this similarity is confirmed by the scores 
on the pretest, the more effective this control becomes. Assuming thst 
these desiderata are approximated for purposes of internal validity, 
wo can regard the design as controlling the main effects of history, 
maturation, testing, and instrumentation . . . (Campbell & Stanley, 
1963, pp. 47-48) 

The current report verifies that the Client Observation Checklist has high 
interrater reliability and is valid by comparison with the TSDS. Since 
difference between facility pretest scores on the COC are small and 
norsignif leant, use of the COC to compare changes at Facility 1 and 2 will be 
an effective technique to evaluate the success of Project ADAPT. 
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Table 1 

Correlations between Three COC Subscales and Six TSDS Activities of Daily Living Items 
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Table 2 

Ratings of Client Observation Checklist Items by Facility 



Facility 1 Facility 2 . x Both (1 & 2) 





No 


rart 


Yes 


No 


Part 


Yes 




No 


Part 


Yes 


BATHING CHECKLIST (sura of 4 


items : 


M r 


2.00; 


M 2 = 


3.73; 


£(28)=1.82, 




.09) 




1. Nec & face 


7 


7 
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4 


7 


4 


2.62,£<.28 


11 


14 


5 


2. Hands & arras 


8 


4 


3 


5 


6 


4 


1.24,£>.50 


13 


10 


7 


3. Chest & groin 


8 


7 


0 


5 


6 


4 


4.77,£<.10 


13 


13 


4 


4. Feet & legs 


11 


4 


0 


5 


7 


3 


6.07,£<.05 


16 


11 


3 


1 

DRESSING CHECKLIST ~ (sura of 


6 items: 


M l 


=3.00; M ? 


=5.73; 


t_(28)=1.72 




. 10) 




1. Selected own clothes 


15 


0 


0 


11 


0 


4 


2 . 60 ,£< . 1 1 


26 


0 


4 


2. Put on underclothes 


9 


4 




5 


3 


7 


4.06,£<.14 


14 


7 


9 


3. Put on clothes 


9 


4 


2 


5 


3 


7 


4.06 ,£<. 14 


14 


7 


9 


4. Zipped, buttoned, etc. 


12 


0 


3 


7 


2 


6 


4.32,£<.12 


19 


2 


9 


5. Put on socks/nylons 


9 


3 


3 


6 


3 


6 


1.60,£>.40 


15 


6 


9 


6. Put on shoes 


8 


0 


7 


6 


3 


6 


3.36,£<.19 


14 


3 


13 


SOCIALIZATION CHECKLIST 1 (sura of 


8 items 




8.40; M 2 =11.00; £(28) 


=1.51 


, £<.15) 


1. "What's your name? 11 


4 


3 


8 


2 


0 


13 


4.86,£<.09 


6 


3 


21 


2. "What's the ai'^e's name?" 


7 


3 


5 


1 


2 


12 


7.53,£<.03 


8 


5 


17 


3. "What... do you do here?" 


5 


3 


7 


2 


0 


13 


6.09,£<.05 


8 


3 


20 


4. Conversation by this time 


13 


0 


2 


9 


0 


6 


1.53,£<.22 


22 


0 


8 


5. Conversation at any point 


9 


0 


6 


8 


0 


7 


0.00 ,£=1.0 


17 


0 


13 


6. Answer anything 


4 


1 


10 


1 


0 


14 


3.47,£<.18 


5 


1 


24 


7. Eye contact 


0 


6 


9 


4 


4 


7 


4.65,£<.10 


4 


10 


16 


8. Conversation with anyone 


7 


0 


8 


7 


1 


7 


1 .07 ,£>. 50 


14 


1 


15 



Note . See Appendix A for an exact statement of the COC items and description of 
the meanings of "No", "Part", and "Yes" ratings for each item. 

*For each subscale CHECKLIST, "Mj" refers to the mean sum score at 
EI\IC facility 1 and "M 2 " refers to the mean sura score at Facility 2, 



APPENDIX A 



A.T. 
B.O.T. 
E.O.T. 



Client Name: 
Client ID: 



CLIENT OBSERVATION CHECKLIST 

Facility 

Rater 



Dfite 



I. BATHING CHECKLIST (sum = 
(washing & drving) 



j>ts.) III. SOCIALIZATION CHECKLIST (sum * 



DCS.) 







No 


Part 


Yes 


1. 


Neck, Mouth, Rest of face 


0 


1 


2 


2. 


Hands & Anns 


0 


1 


2 


3. 


Chest & Groin 


0 


1 


2 


4. 


Feet & Legs 


0 


1 


2 



II. DRESSING CHECKLIST (sum ■ 

1. Selected own clothes 

2. Put on underclothes 

3. Put on clothes 
4 



Zipped, Buttoned, 
Snapped, Hooked 

Put on socks/nylons 
Put on shoes 



j)ts.) 
0 
0 
0 

0 
0 
0 



1 
1 



1 
1 
1 



2 
2 
2 

2 
2 
2 



RATER INTRODUCES SELF XoJ?artjLes 

1. "What's your name?" 0 12 

2. "What's the aide's name?" 0 1 2 



"What sort of things do 
you do here?" 

SHORT PAUSE 



0 



4. has the client initiated 
any conversation vith you 

by this time? 0 1 r i 

AFTER THE FIRST 3 SET QUESTIONS YOU MAY 
CONVERSE WITH THE CLIENT ON ANY TOPIC, 
COMPLETE THE FOLLOWING AFTER THE CLIENT 
HAS BEEN DRESSED. 

5. Did the client initiate 
conversation with you at 

any point? 0 1 ; 

6. Did the client ever answer 
anything you asked? 0 1 \ 

7. Was the client willing to 

have eye contact with you?0 1 

8. Did the client initiate 
conversation with anyone? 0 1 



COMMENTS: 



li 
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STANDARDS FOR CLIENT OBSERVATION _CHECKLIST 

ABBREVIATIO NS: A.T. - Arrival Time - On the first client observation check list, nar' the 

time you arrived at the hoirr. On successive c>*»c*l fs, 
mark the nme your last chick] 1st v^s c<>mi» J vt i , 

B.O.T. = Beginning Observat io n Time - The time you bipfn the ohn I Mm 

K.0.7. * E nd 1 n g Obb ervn 1 1 on Time - The ( Inn t hr c In < k Mm I i imuj' ' • f« ( ' 

I. BATHING CHECKLIST 

NO * unable or refused to perform the task. 
PART 3 required assistance or verbal cues to perform the task. 
YES = independently performed the task. 

PLEASE NOTE : IF A SPECIAL CLEANING PROBLEM EXISTS, (i.e., COLOSTOMY, FACTaL WOUND, ETC.) 

THE CLIENT SHOULD BE ABLE TO CLEAN APPROPRIATELY IN ORDER TO RATE AS "YES." 

*Specdfic examples for " PART 11 

1. missed neck, mouth or rest of face 3. missed chest or groin 

2. missed a hand or arm 4. missed a foot or leg 

II. DRESSING CHECKLIST 

NO * unable or refused to perform the task. 
PART * required assistance or verbal cues to perform the task or started, but did not 

complete the task.* (see below) 
YES * independently performed the task. 

^Specific examples for "PART". 

1. can choose one or more appropriate items of clothing, or chooses all daily wear, tut 
not suitable to the climate. 

2. inside out or backwards 

3. inside out or backwards 

A. buttons or snaps wrong hole 

5. inside out 

6. wrong feet or can not tie laces 

III. SOCIALIZATION CHECKLIST 

Rating should be done according to how the client responds to the interview during the 
bathing and dressing observations. 

QUESTIONS 1, 2, 3 

NO * ignocrd, unable or refused to respond. 
PART « begins a response but stops, or responds to a verbal prompt by other staff, or 
communicates nonverbally (ie, headshake), or with non-English vocalization 
(ie, grunt) 

YES ■ any coherent response even if answer is incorrect 
QUESTION 7 

NO * no eye contact. 

PART = glanced at rater one or more times, but did not maintain eye contact for longer than a 
second. 

YES * maintained eye contact for longer than a second. 

DEFINITION: Initiate conversation - any coherent, logical sequence of two or more words requiring 
a response from someone. 
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